Going Wider: Recurrent Neural Network With Parallel Cells

Recurrent Neural Network (RNN) has been widely applied for sequence modeling. In RNN, the hidden states at current step are full connected to those at previous step, thus the influence from less related features at previous step may potentially decrease model’s learning ability. We propose a simple technique called parallel cells (PCs) to enhance the learning ability of Recurrent Neural Network (RNN). In each layer, we run multiple small RNN cells rather than one single large cell. In this paper, we evaluate PCs on 2 tasks. On language modeling task on PTB (Penn Tree Bank), our model outperforms state of art models by decreasing perplexity from 78.6 to 75.3. On Chinese-English translation task, our model increases BLEU score for 0.39 points than baseline model. 1.Introduction Recurrent neural network (RNN) has been one of the most powerful sequence models in natural language processing. Many important applications achieve state of the art performance with RNN, including language modeling(Mikolov et al.,2010; Zaremba et al., 2014 ), machine translation (Luong and Manning, 2016; Wu et al., 2016) and so on. The features learned by RNN are stored in the hidden states. At each time step, the cell extracts features from data and updates its hidden states. The left side of figure 1 concisely shows the transition of hidden states in naïve RNN. We can see that all units at the previous step are fully connected to all units at the current step. Thus, each pair of features can affect each other. Such design is not reality because many features are not that related. The influence between unrelated features may harm the learning ability of models. We can expect learning models automatically set the weight of all unnecessary connections to zero. However, in practice, because the data size is limited and the algorithms are not that strong, these unrelated connections will harm the learning ability. For example, that is why we have to do feature selections before training. To address the problem, many successful neural models benefit from replacing global connection with local connection. For example, Long Short-Term Memory(LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units(GRU) (Cho et al., 2014) have been the most popular RNN cells. In the core, such models use gates to control the data flow, allow part of connections to be activated. Another example is Convolution Neural Networks(CNN)(Lecun et al, 1998), one of the most successful models in deep learning nowadays. CNN uses local receptive fields to extract features from previous feature map. With local receptive fields, neurons can extract elementary visual features such as oriented edges, end-points, corners.